Minimizing impact of migrating virtual services

ABSTRACT

The present disclosure relates to systems, methods, and computer readable media that utilize a low-impact live-migration system to reduce unfavorable impacts caused as a result of live-migrating computing containers between physical server devices of a cloud computing system. For example, systems disclosed herein evaluates characteristics of computing containers on server devices to determine a predicted unfavorable impact of live-migrating the computing containers between the server devices. Based on the predicted impact, the systems disclosed herein can selectively identify which computing containers to live-migrate as well as carry out live-migration of the select computing containers in such a way the significantly reduces unfavorable impacts to a customer or client device associated with the computing containers.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.16/160,756, filed on Oct. 14, 2018, the entirety of which isincorporated herein by reference.

BACKGROUND

A cloud computing system refers to a collection of computing devices onwhich data can be remotely stored and accessed. For example, moderncloud computing infrastructures often include a collection of physicalserver devices organized in a hierarchical structure including computingzones, clusters, virtual local area networks (VLANs), racks, faultdomains, etc. Cloud computing systems often make use of different typesof virtual services (e.g., computing containers, virtual machines) thatprovide remote storage and computing functionality to various clients orcustomers. These virtual services can be hosted by respective servernodes on a cloud computing system.

As cloud computing systems have grown in popularity and increased incomplexity, managing different types of services and structures across adistributed network of computing devices has become increasinglydifficult. Indeed, many conventional cloud computing systems enhancestorage space and processing capabilities of distributed computingdevices by live-migrating computing containers, virtual machines, orother virtual services from one server node to another server nodewithin the cloud computing system. Live-migrating virtual services canoccur under a variety of circumstances including, by way of example,emptying a server node for an operating system (OS) update, balancingstorage capacity between server nodes to make room for more allocatablecapacity across multiple server nodes, and avoiding disconnection fromcomputing structures as a result of predicted server failure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example environment of a network of a distributedcomputing system including a low-impact live-migration system inaccordance with one or more implementations.

FIG. 2 illustrates an example low-impact live-migration system formigrating one or more virtual machines between server nodes of anexample node cluster in accordance with one or more implementations.

FIG. 3 illustrates an example framework for selectively live-migratingvirtual machines to minimize impact in accordance with one or moreimplementations.

FIG. 4 illustrates an example implementation in which virtual machinesare selectively live-migrated between nodes of a node cluster.

FIG. 5 illustrates another example implementation in which virtualmachines are selectively live-migrated between nodes of a node cluster.

FIG. 6 illustrates an example process for live-migrating a virtualmachine in accordance with one or more implementations.

FIG. 7 illustrates an example of a method for live-migrating one or morevirtual machines from a select server node in accordance with thepresent disclosure.

FIG. 8 illustrates certain components that may be included with acomputer system.

FIG. 9 illustrates certain components that may be included within acomputer system.

DETAILED DESCRIPTION

The present disclosure is generally related to a low-impactlive-migration system implemented on a cloud computing system (e.g.,distributed network of computing devices) that minimizes unfavorableimpacts caused by live-migrating virtual machines, virtual computingcontainers (or simply “computing containers”), or other virtual servicesbetween server devices (e.g., server nodes) of the cloud computingsystem. In particular, as will be described in further detail below, thelow-impact live-migration system evaluates characteristics of aplurality of virtual services on one or more server devices to determinea predicted unfavorable impact of live-migrating the virtual servicesbetween server devices. Based on the predicted impact of live-migratingthe respective services, the low-impact live-migration system canselectively identify one or more computing containers and/or virtualmachines to live-migrate as well as schedule a time for live-migrationbased on the predicted impact associated with live-migrating therespective virtual service(s) at a specific time. In addition, thelow-impact live-migration system can initiate live-migration of theselect services in accordance with one or more implementations describedherein.

Implementations of the present disclosure provide benefits and/or solveone or more problems associated with live-migrating computing containersand/or virtual machines between server devices of a cloud computingsystem. In particular, live-migrating containers and virtual machinesbetween server devices often unfavorably impacts customer servicesand/or connections between client devices and the computing containersor virtual machines. In addition, live-migrating often causes blackouts,brownouts, or other interruptions that negatively impact a service orconnection between a client device and the cloud computing system. As aresult, clients can become frustrated as applications slow, operate atreduced throughput or increased latency, or become temporarilydisconnected from the cloud computing system while computing containersor virtual machines that provide the various applications arelive-migrated between server nodes.

As will be described in further detail below, the low-impactlive-migration system can avoid many of the unfavorable impacts commonlycaused by live-migration in a variety of ways. For example, byevaluating characteristics of different virtual services, the low-impactlive-migration system can distinguish between computing containers orvirtual machines that may have different levels of sensitivity tolive-migration. For instance, a virtual machine for an online gamingapplication may have a high sensitivity to live-migration where even ashort blackout (e.g., 1-2 seconds) or brownout causes the client deviceto disconnect from an active game. Alternatively, a virtual machine thatprovides large-scale computing applications may have a higher toleranceto a short blackout or brownout, ultimately having a very small impactfrom the perspective of a customer.

In addition to live-migration sensitivity, the low-impact live-migrationsystem can further identify other characteristics of computingcontainers and virtual machines related to a potential impact ofperforming live-migration at a particular time. For example, thelow-impact live-migration system can identify usage patterns associatedwith a virtual machine to determine a time of day that live-migratingthe virtual machine may have a higher or lower impact on a customer. Asanother example, the low-impact live-migration system can determine asize or input/output (I/O) activity of the virtual machine and howlive-migration may impact providing service to a client device whilemoving containers between server devices. As a further example, thelow-impact live-migration system can identify a projected lifetime of avirtual machine to determine whether the virtual machine is set toexpire or discontinue operation in the near future. Indeed, thelow-impact live-migration system can identify any number of computingcontainer or virtual machine characteristics that may influence animpact of live-migrating a virtual service from one device to another.

Based on the identified computing container or virtual machinecharacteristics, the low-impact live-migration system can apply orotherwise implement a prediction engine trained to determine a projectedimpact of performing live-migration. In particular, as will be describedin further detail below, the low-impact live-migration system can applythe prediction engine to any number of computing containers and/orvirtual machines to determine impact scores associated with a projectedimpact of live-migrating one or more of the virtual service(s) betweendevices on the cloud computing system. The prediction engine can be usedto determine a migration impact in a variety of ways, examples of whichwill be described in further detail below.

Based on the determined impact of live-migrating the computingcontainers, the low-impact live-migration system can selectivelyidentify computing containers and/or virtual machines forlive-migration. In particular, the low-impact live-migration system canidentify a subset of computing containers or virtual machines from alarger collection of virtual services having lower impact scores andinitiate live-migration of the identified subset. In this way, thelow-impact live-migration system can selectively identify those virtualservices that will have a lower impact on customers, which cansignificantly decrease and even avoid some of the negative effects oflive-migrating computing containers indiscriminately (e.g., withoutconsidering timing of live-migration or various containercharacteristics).

The low-impact live-migration system provides a number of advantagesover conventional systems for live-migrating services between servernodes. In particular, the low-impact live-migration system canselectively live-migrate one or more computing containers or virtualmachines to improve overall performance and storage capacity of thedistributed computing system. As an example, the low-impactlive-migration system can trigger live-migrating one or more virtualmachines in order to maintain a desired number of empty server nodes ona cluster of server nodes. Indeed, by maintaining a certain number(e.g., a threshold minimum or desired number) of empty server nodes, thelow-impact live-migration system can increase flexibility of a clusterof server devices, improve fault tolerance, as well as increase aquantity of allocatable computing space on a cluster of server nodes.

In addition, the low-impact live-migration system can triggerlive-migration to improve performance of the cloud computing system inother ways. For example, the low-impact live-migration system cantrigger live-migration to de-fragment capacity of one or more serverdevices, thereby increasing allocatable space across a cluster of serverdevices. As another example, the low-impact live-migration system canpredict failure of a server node and generate a migration schedule thattimely live-migrates virtual machines on the server node prior tofailure while minimizing an unfavorable impact of the live-migration. Asa further example, the low-impact live-migration system can live-migratevirtual machines from a server device to enact an OS update withoutinterrupting a connection to the virtual machines on the server devicewhile updating the OS.

As illustrated in the foregoing discussion, the present disclosureutilizes a variety of terms to describe features and advantages of thelow-impact live-migration system. Additional detail is now providedregarding the meaning of such terms. For instance, as user herein, a“cloud computing system” refers to a network of connected computingdevices that provide various services to client devices. For instance,as mentioned above, a distributed computing system can include acollection of physical server devices (e.g., server nodes) organized ina hierarchical structure including computing zones, clusters, virtuallocal area networks (VLANs), racks, fault domains, etc. In addition, itwill be understood that while one or more specific examples andimplementations described herein relate specifically to “clusters” ofserver nodes, features and functionality described in connection withone or more node clusters described herein can similarly relate toracks, fault domains, or other hierarchical structures of physicalserver devices. The cloud computing system may refer to a private orpublic cloud computing system.

As used herein, a “virtual service” refers to a service provided by acloud computing system. A virtual service may provide an operatingsystem, which

As used herein, a “computing container” or “virtual computing container”refers to a virtual service or layer on a server node of a cloudcomputing system that provides access to a storage space and/or softwareapplication hosted by the cloud computing system. Computing containerscan provide services to any number of containerized applications on acloud computing system.

As used herein, a “virtual machine” refers to an emulation of a computersystem on a server node that provides functionality of one or moreapplications on the cloud computing system. Virtual machines can providefunctionality needed to execute one or more operating systems. Inaddition, virtual machines can make use of hypervisors on processors ofserver devices that support virtual replication of hardware. It will beunderstood that while one or more specific examples and implementationsdescribed herein relate specifically to virtual machines, features andfunctionality described in connection with identifying virtual machinecharacteristics and live-migrating virtual machines can similarly applyto other types of computing containers.

As used herein, “live-migration” refers to a process whereby data from aserver node is migrated to another server node (e.g., a destinationserver node). For example, live-migration may refer specifically tomigrating a virtual machine from one device to another device whileproviding full or limited connectivity to a client device to the virtualmachine. As mentioned above, live-migration may occur in a variety ofways and may be performed under various circumstances to improve storagecapacity and functionality of a cloud computing system.

As used herein, “migration impact” or “impact” refer interchangeably toa predicted impact of accessing a computing container as a result of aninterruption in connectivity between a client device and a virtualmachine hosted by one or more server nodes. For example, an impact mayrefer to a metric that indicates a time or duration that a client deviceis disconnected from or experiences limited connectivity to anapplication provided by a virtual machine. In one or more embodiments,an impact is measured by an impact score that indicates a level ofimpact ranging from a low impact (e.g., a less noticeable impact to acustomer) to a high impact (e.g., a more noticeable impact to acustomer). In addition, as will be described in further detail below, animpact may vary based on a type of application, a size of a virtualmachine, or other characteristics of a virtual machine in accordancewith one or more embodiments described herein.

Additional detail will now be provided regarding the low-impactlive-migration system in relation to illustrative figures portrayingexample implementations. For example, FIG. 1 illustrates an exampleenvironment 100 for managing live-migration of virtual machines betweenserver nodes of a cloud computing system 101. As shown in FIG. 1, theenvironment 100 includes a cloud computing system 101. The cloudcomputing system 101 includes one or more server device(s) 102 includinga low-impact live-migration system 104. The cloud computing systemfurther includes a plurality of node clusters 106 a-n. Each of the nodeclusters 106 a-n include multiple server nodes 108 a-n, which includeone or more virtual machines 110 a-n thereon. While FIG. 1 illustrates aspecific arrangement of server device(s) 102, node clusters 106 a-n,server nodes 108 a-n, and virtual machines 110 a-n, the cloud computingsystem 101 can include any number of node clusters (or otherhierarchical structure of devices), which may include any number ofserver nodes, each of which may include zero, one, or multiple virtualmachines, computing containers and/or other types of virtual servicesimplemented thereon.

As further shown, the environment 100 includes a plurality of clientdevices 112 a-n in communication with the cloud computing system 101(e.g., in communication with different server nodes 108 a-n) via anetwork 114. The client devices 112 a-n may refer to various types ofcomputing devices including, by way of example, mobile devices, desktopcomputers, or other types of computing devices. In addition, the networkmay include one or multiple networks that use one or more communicationplatforms or technologies for transmitting data. For example, thenetwork 114 may include the Internet or other data link that enablestransport of electronic data between respective client devices anddevices of the cloud computing system 101.

In one or more implementations, the virtual machines 110 a-n correspondto one or more customers and provide access to storage space and/orapplications hosted by the server nodes 108 a-n. For example, a virtualmachine may provide access to a large-scale computation application to aclient device 112 a (or multiple client devices). As another example, adifferent virtual machine on the same server node or a different servernode may provide access to a gaming application to a second clientdevice 112 b (or multiple client devices).

As will be described in further detail below, the low-impactlive-migration system 104 can facilitate live-migration of one or morevirtual machines 110 a-n between server nodes 108 a-n across the cloudcomputing system 101. As an example, the low-impact live-migrationsystem 104 can live-migrate one or more virtual machines from a servernode within a first node cluster 106 a to another server node within thesame node cluster 106 a. Alternatively, the low-impact live-migrationsystem 104 can live-migrate one or more virtual machines from a servernode within the first node cluster 106 a to another server node within asecond node cluster 106 b.

More specifically, the low-impact live-migration system 104 canfacilitate live-migration of virtual machines between server nodes byselectively identifying virtual machines for live-migration as well asidentifying a time of live-migration and a destination server node thatavoids or otherwise reduces an impact of the live-migration to one ormore of the client devices 112 a-n having access to the virtualmachine(s). As mentioned above, by selectively identifying virtualmachines and strategically timing migration of the virtual machinesbetween server nodes, the low-impact live-migration system 104 cansignificantly reduce connection and accessibility issues (e.g.,different types of impact) that live-migrating the virtual machines canhave on the client devices 112 a-n.

Additional detail with regard to implementing live-migration of avirtual machine between server nodes within a node cluster will now bedescribed in connection with FIG. 2. Similar features can similarlyapply to live-migrating virtual machines between server nodes ofdifferent node clusters. As shown in FIG. 2, the low-impactlive-migration system 104 may be implemented on a central resourcedevice or system of devices to provide a number of features andfunctionalities described here. In particular, the low-impactlive-migration system includes a data collection engine 202, an impactprediction engine 204, and a migration engine 206. As further shown, anode cluster 208 includes server nodes 210 a-c including virtualmachines 212 a-f implemented thereon. The node cluster 208 additionallyincludes a number of empty nodes 214 that do not include one or morevirtual machines operating thereon. The node cluster 208 may be anexample of any of the node clusters 108 a-n described in connection withFIG. 1.

While FIG. 2 illustrates an example arrangement of nodes includingserver nodes 210 a-c including one or multiple virtual machines 212 a-fthereon in addition to a number of empty nodes 214, the node cluster 208may include any number of server nodes 210 a-c, each of which mayinclude one or multiple virtual machines. In addition, the node cluster208 can include any number of empty nodes 214.

As mentioned above, the low-impact live-migration system 104 includes adata collection engine 202. The data collection engine 202 can collect,obtain, or otherwise access data associated with the virtual machines212 a-f. The data collection engine 202 may further identify any numberof virtual machine characteristics associated with the virtual machines212 a-f. Virtual machine characteristics may include a number ofdifferent features or functionalities of a virtual machine in additionto characteristics of client devices and/or customer(s) associated withthe virtual machine.

As an example, the data collection engine 202 can identify usagecharacteristics for a virtual machine. This may include a classificationof time periods for which usage of the virtual machine is heavy orlight. For example, a virtual machine that serves a number of devicesused primarily for work purposes may experience heavy use during typicalwork hours while experiencing very light use at night or on weekends.Alternatively, a virtual machine that serves online gamers mayexperience heavy use during the evening, at night, or on weekends whileexperiencing lighter use during the morning, afternoon, or on weekdays.Similarly, virtual machines that serve clients across different timezones may have different usage characteristics for different ranges oftime.

As another example, the data collection engine 202 can identify alifetime characteristic for a virtual machine. This may includeidentifying a time that the virtual machine is set to expire or aduration of time that the virtual machine is scheduled to operate. Aswill be described in further detail below, this may influence a decisionto live-migrate a virtual machine within the near future or, rather,simply wait until the virtual machine expires rather than live-migratethe virtual machine at all.

The data collection engine 202 can additionally identify a sensitivitycharacteristic for the virtual machine associated with a sensitivity ofthe virtual machine with respect to experiencing a disconnection and/orslowed connection with a client device. In particular, as mentionedabove, different virtual machines may host or provide access todifferent applications, which may differ significantly in sensitivity tolive-migration. For example, the data collection engine 202 can identifywhether a virtual machine hosts a gaming application that is highlysensitive to outages (e.g., brownouts, blackouts) and may be incapableof tolerating a one or two second disconnection between a client deviceand the virtual machine. Alternatively, the data collection engine 202can identify whether a virtual machine supports simple data storagefeatures, which may have lower sensitivity to a temporary disconnectionand may be better capable of tolerating multiple seconds of an outagewithout significantly impacting a customer experience.

The data collection engine 202 can further identify memorycharacteristics associated with memory access or utilization of memoryon the server node by the virtual machine. This may include a type ofprocessor implemented by the server node corresponding to the virtualmachine, a type of memory access pattern, capabilities of the virtualmachine or associated server node, or other characteristics associatedwith memory access or utilization of memory by the virtual machine.

In one or more implementations, the data collection engine 202identifies customer or client characteristics associated with one ormore client devices and/or customers that have access to the virtualmachine. For example, where customers may vary from trial customers orguests having temporary access to a virtual machine to paying customersor high priority customers that frequently use one or multipleapplications hosted by a virtual machine, the data collection engine 202may identify one or more characteristics associated with the differenttypes of customers.

As further shown in FIG. 2, the low-impact live-migration system 104includes an impact prediction engine 204. The impact prediction engine204 can evaluate some or all of the identified virtual machinecharacteristics to estimate a predicted impact that migrating a virtualmachine will have on a customer. For example, the impact predictionengine 204 may evaluate a set of one or multiple types of virtualmachine characteristics identified for a virtual machine to determine anestimated measure or level of impact (e.g., ranging from no impact to ahigh level of impact) that live-migrating the virtual machine from oneserver node to another server node will have on a customer. In one ormore embodiments, the impact prediction engine 204 determines anestimated level of impact for each virtual machine on a server node oracross multiple server nodes of the node cluster 208.

In one or more embodiments, the impact prediction engine 204 determinesthe predicted impact by determining an impact score for one or more ofthe virtual machines. The impact score indicates a predicted measure ofimpact ranging from zero impact to a high level of impact. As will bediscussed in further detail below, the impact prediction engine 204 canincludes an algorithm or model (e.g., a machine learning model) trainedto determine a predicted impact score based on a combination ofdifferent virtual machine characteristics. The impact score may refer toa number, category, or other measure from a range of numbers where alower number or category from the range indicates a low measure ofimpact while a higher number or category from the range indicates a highmeasure of impact.

As will be described in further detail below, the impact predictionengine 204 can include one or more algorithms or models that affect animpact score based on different types of virtual machinecharacteristics. For instance, the impact prediction engine 204 canconsider different types of virtual machine characteristics usingdifferent models or based on different weighting factors to determine animpact score for a virtual machine having a set of virtual machinecharacteristics. Addition detail with regard to different types ofmodels that may be included within the impact prediction engine 204 isprovided in further detail below (e.g., in connection with FIG. 3).

As further shown in FIG. 2, the low-impact live-migration system 104includes a migration engine 206. The migration engine 206 mayselectively identify one or more virtual machines to live-migratebetween server nodes of the node cluster 208. In one or moreimplementations, the migration engine 206 identifies a subset of virtualmachines from a collection of virtual machines to live-migrate from oneserver node to another server node. As an example, the migration engine206 can identify virtual machines 212 a-b on a first server node 210 ato live-migrate to a second server node 210 b based on a determinationthat live-migrating the first and second virtual machines 212 a-b willhave a lower projected impact on a customer than migrating the virtualmachines 212 c-e on the second server node 210 b.

As will be explained in further detail below, the migration engine 206can selectively identify virtual machines for migration to improveoperation of server nodes on the node cluster 208. For example, themigration engine 206 may selectively identify virtual machines formigration from a particular server node or node cluster to free up spaceon individual server nodes and/or across multiple server nodes of thenode cluster. Accordingly, the migration engine 206 can selectivelyidentify a subset of virtual machines from a collection of virtualmachines having the lowest impact scores to live-migrate. This can freeup space on the node cluster or on an individual server node while alsominimizing an unfavorable impact of the live-migration process.

As another example, the migration engine 206 may identify sets ofvirtual machines on respective server nodes to live-migrate to ensurethat the node cluster 208 has a desired number (e.g., a minimumthreshold or other predetermined number) of empty nodes on the nodecluster 208. Accordingly, in one or more implementations, the migrationengine 206 identifies a server node including a set of virtual machineshaving lower impact scores (e.g., a lower combined impact score) than adifferent set of virtual machines on another server node. In this way,the migration engine 206 can selectively evacuate an entire server nodedetermined to have the lowest migration impact relative to other servernodes on the node cluster.

Accordingly, the migration engine 206 can selectively identify virtualmachines to migrate to both minimize unfavorable impacts oflive-migration and redistribute virtual machines across server nodes ofthe node cluster 208 in such a way to improve performance of the servernodes 210 a-c on the node cluster 208. Additional examples ofselectively identifying virtual machines to accommodate variousapplications of the server node are provided in connection with FIGS.4-5 below.

FIG. 3 illustrates an example framework for selectively identifyingvirtual machines for migration based on impact scores determined for thevirtual machines implemented by the low-impact live-migration system104. In particular, as shown in FIG. 3, the framework includes the datacollection engine 202, impact prediction engine 204, and migrationengine 206 that the low-impact live-migration system 104 utilizes tocollect virtual machine characteristics, determine impact scores basedon the virtual machine characteristics, and further identify virtualmachines for live-migration based on the impact scores.

In particular, as shown in FIG. 3, the data collection engine 202receives, collects, or otherwise accesses virtual machine data includingany data associated with one or more virtual machines. This may includeinformation about a virtual machine, information about a client deviceor customer associated with the virtual machine, and/or informationabout a server node and/or node cluster on which the virtual machineoperates.

The data collection engine 202 may evaluate the data to identify one ormore virtual machine characteristics associated with the virtualmachine(s). For instance, in one or more embodiments, the datacollection engine analyzes the virtual machine data to identify one ormore discrete signals or characteristics of the virtual machines thatmay be associated with or influence a projected impact of live-migratingthe virtual machines. This may include identifying a size of a virtualmachine, memory access patterns of the virtual machine, sensitivity ofthe virtual machine, a priority of a customer associated with thevirtual machine, or any other virtual machine characteristic(s).

The data collection engine 202 can provide any number of virtual machinecharacteristics to the impact prediction engine 204, which can determineimpact scores for any number of virtual machines associated with thevirtual machine characteristics. The impact prediction engine 204 candetermine impact scores based on the virtual machine characteristics inaccordance with one or more embodiments described above. In addition, asshown in FIG. 3, the impact prediction engine 204 may include one ormore different types of prediction engines trained to determine discreteimpact scores associated with different types of predicted impacts thatlive-migrating a virtual machine may cause.

For example, as shown in FIG. 3, the impact prediction engine 204includes a blackout prediction engine 302, a brownout prediction engine304, a lifetime prediction engine 306, a sensitivity prediction engine308, and a resource utilization prediction engine 310. As will bediscussed further, each of the different types of prediction engines302-310 may predict an impact score associated with a different type ofcustomer impact. In addition, these prediction engines 302-310 aredescribed by way of example, and one or more embodiments of the impactprediction engine 204 may include additional types of prediction enginestrained to determine prediction scores associated with different typesof customer impacts (e.g., based on the virtual machine characteristicsprovided by the data collection engine 202).

As mentioned above, the impact prediction engine 204 includes a blackoutprediction engine 302. The blackout prediction engine 302 may include analgorithm or prediction model trained to determine a predicted blackouttime for a virtual machine based on a corresponding set of virtualmachine characteristics associated with the virtual machine. Forexample, the blackout prediction engine 302 may determine an estimatedtime-period that a virtual machine will become disconnected from aclient device as a result of live-migrating the virtual machine from oneserver node to another server node on a node cluster. In addition, or asan alternative to determining an estimated time of disconnection, theblackout prediction engine 302 may determine an estimated period of timethat the virtual machine is frozen during which no codes or instructionsare executed. The predicted blackout time may range anywhere from a verysmall time-duration (e.g., 0 seconds) corresponding to a low impactscore to a high time-duration (e.g., 30+ seconds) corresponding to ahigh impact score.

The blackout prediction engine 302 can determine the predicted impactscore in a number of ways and based on any combination of virtualmachine characteristics. For example, one or more implementations of theblackout prediction engine 302 can include an algorithm or machinelearning model trained to classify or categorize the virtual machineinto one of a predetermined range of blackout times. In particular, theblackout prediction engine 302 can apply a machine learning model or analgorithm to a set of virtual machine characteristics to predict ablackout time that the virtual machine is expected to experience andclassify the virtual machine based on one of a plurality of blackouttime ranges.

As a non-limiting example, the blackout prediction engine 302 mayutilize a Random Forest learning algorithm trained to analyze a set ofvirtual machine characteristics or features to predict a blackoutcategory corresponding to a discrete range of blackout times that thevirtual machine is predicted to experience as a result oflive-migration. In determining the predicted blackout time, the RandomForest learning algorithm may analyze characteristics including, by wayof example, cluster characteristics, OS characteristics, size of thevirtual machine, size of a family of virtual machines, customer usagepatterns (e.g., heavy or light usage on weekdays, at night, or duringspecific times of the day), a number of cores of a processor operatingin connection with the virtual machine and/or on a corresponding servernode, or any other characteristics for which the blackout predictionengine 302 is trained to utilize in predicting a blackout time that thevirtual machine will experience as a result of live-migrating thevirtual machine between server nodes.

In one or more embodiments, the blackout prediction engine 302 generatesan impact score including a classification of the virtual machine withina category of blackout times. For example, the blackout predictionengine 302 can utilize the Random Forest learning algorithm or otherprediction model to determine a predicted blackout time for the virtualmachine and categorize the virtual machine into a bucket or othercategory of blackout time ranges. For instance, the blackout predictionengine 302 can classify the virtual machine into one of a plurality ofcategories or buckets using a log linear scale. An example of a loglinear scale of categories includes a first bucket for a predictedblackout time of less than 0.1 seconds, a second bucket for a predictedblackout time between 0.1 seconds and 1.0 seconds, a third bucket for apredicted blackout time between 1.0 seconds and 5.0 seconds, a fourthbucket for a predicted blackout time between 5.0 and 10.0 seconds, and afifth bucket for any predicted blackout time greater than 10.0 seconds.Other ranges of predicted blackout times may similarly apply.

Utilizing a machine learning model trained to predict categories ofblackout times using a log linear scale has been found to yield accuratepredictions of estimated blackout times. In particular, tests thatutilize a Random Forest learning algorithm that determines a predictedcategory of blackout times on a log linear scale (e.g., according to thecategories just mentioned in connection with the blackout predictionengine 302) based on virtual machine characteristics including clustersize, OS type, virtual machine size, a count of processing cores,virtual machine memory type, and other virtual machine characteristicshave been found to predict a blackout time category with at least 89%accuracy.

As mentioned above, the impact prediction engine 204 may include abrownout prediction engine 304. The brownout prediction engine 304 mayinclude a prediction model or algorithm to determine a predictedbrownout time for a virtual machine based on a corresponding set ofvirtual machine characteristics associated with the virtual machine. Forexample, the brownout prediction engine 304 may determine an estimatedduration of time that a virtual machine will experience a slowconnection (e.g., a slower than normal connection under typicaloperating conditions) as a result of live-migrating the virtual machinefrom one server node to another server node on the cloud computingsystem 101. As another example, the brownout prediction engine 304 maydetermine a period of time during which the virtual machine will providelimited functionality (e.g., reduced throughout, increased latency,limited I/O capabilities, or other functions otherwise available whenoperating at full functionality), The predicted brownout time may rangefrom a very small duration of time (e.g., less than one minute)corresponding to a low impact score to a high duration of time (e.g., anhour or more) corresponding to a high impact score.

Similar to the blackout prediction engine 302, the brownout predictionengine 304 can determine the predicted impact score in a number of waysand based on any combination of virtual machine characteristics. Forexample, one implementation of the brownout prediction engine 304 caninclude an algorithm or machine learning model trained to classify orcategorize the virtual machine into one of a predetermined range ofbrownout times. In particular, the brownout prediction engine 302 canapply a machine learning model or an algorithm to a set of virtualmachine characteristics to predict a brownout time that the virtualmachine is expected to experience and classify the virtual machine basedon one of a plurality of brownout time ranges.

As a non-limiting example, the brownout prediction engine 304 mayutilize a Random Forest learning algorithm trained to analyze a set ofvirtual machine characteristics or features to predict a brownoutcategory corresponding to a discrete range of brownout times that thevirtual machine is predicted to experience as a result oflive-migration. In determining the predicted brownout time, the RandomForest learning algorithm may analyze characteristics including similarcharacteristics as described above in connection with the blackoutprediction engine 302. Alternatively, the brownout prediction engine 304may consider other characteristics that may have an impact on brownouttimes.

In one or more embodiments, the brownout prediction engine 304 generatesan impact score including a classification of the virtual machine withina category of blackout times. For example, the brownout predictionengine 304 can utilize the Random Forest learning algorithm or otherprediction model to determine a predicted brownout time for the virtualmachine and categorize the virtual machine into a bucket or othercategory of brownout time ranges. For instance, the brownout predictionengine 304 can classify the virtual machine into one of a plurality ofcategories or buckets using a log linear scale. An example of a loglinear scale of categories includes a first bucket for a predictedbrownout time of less than 1.0 minutes, a second bucket for a predictedbrownout time between 1.0 minutes and 5.0 minutes, a third bucket for apredicted brownout time between 5.0 minutes and 10.0 minutes, a fourthbucket for a predicted brownout time between 10.0 minutes and 20.0minutes, a fifth bucket for a predicted brownout time between 20.0minutes and 50.0 minutes, and a sixth bucket for a predicted brownouttime greater than 50.0 minutes. Other ranges of predicted brownout timesmay similarly apply.

Utilizing a machine learning model trained to predict buckets orcategories of brownout times using a log linear scale has been found toyield accurate predictions of estimated brownout times. In particular,tests that utilize a Random Forest learning algorithm that determines apredicted category of blackout times on a log linear scale (e.g.,according to the categories just mentioned with regard to the brownoutprediction engine 304) based on virtual machine characteristicsincluding cluster size, OS type, virtual machine size, a count ofprocessing cores, virtual machine memory type, and other virtual machinecharacteristics have been found to be able to predict a brownout timecategory with at least 91% accuracy.

As mentioned above, the impact prediction engine 204 includes a lifetimeprediction engine 306. The lifetime prediction engine 306 may include aneural network or other machine learning model trained to identify apredicted lifetime as well as determine an impact score based on timethat a given virtual machine is set to expire or disconnect from thecloud computing system 101. For instance, where a virtual machine has alimited lifespan corresponding to a subscription, a trial period, orother virtual machine characteristic associated with a time that thevirtual machine will disconnect from the cloud computing system 101, theimpact prediction engine 204 may determine an impact score thatindicates the scheduled disconnection.

As an example, where a virtual machine is set to disconnect within 1-2days (or prior to a time when the virtual machine would need tolive-migrate to a destination node), and where the reason forlive-migrating the virtual machine is not particularly urgent, thelifetime prediction engine 306 may generate a high impact score or otherindication that the migration engine 206 would use to determine not tolive-migrate the virtual machine. Indeed, where the virtual machine isset to expire in the near future, live-migrating the virtual machine mayserver a very limited purpose, and the migration engine 206 maydetermine that the least impact to the customer would be achieved bysimply not migrating the virtual machine based on the projected lifetimeof the virtual machine.

As further shown, the impact prediction engine 204 includes asensitivity prediction engine 308. The sensitivity prediction engine 308may include a model or algorithm to predict an impact score based on apredicted sensitivity of the virtual machine and/or applications (orspecific code) running thereon. For example, where virtual machinecharacteristics indicate a high tolerance to blackouts and/or brownouts,the sensitivity prediction engine 308 may generate a low impact scoreindicating that live-migrating the virtual machine, even where blackoutsor brownouts may occur, will have a minimal impact on a customerexperience. Alternatively, where a virtual machine is predicted to havea low tolerance for blackouts and/or brownouts, the sensitivityprediction engine 308 may generate a high impact score indicating thateven a small or trivial blackout and/or brownout will have a significantimpact on the customer experience. As another example, an owner,administrator, or customer associated with a virtual machine may markthe virtual machine as sensitive or otherwise risk averse, which thesensitivity prediction engine 204 may use in determined a predictedsensitivity for the virtual machine.

The impact prediction engine 204 may further include a resourceutilization prediction engine 310. The resource utilization predictionengine 310 may generate an impact score based on memory accesscharacteristics or computing architecture of the virtual machine thatmay have an additional influence on a projected impact of live-migratingthe virtual machine.

In one or more embodiments, each of the different types of predictionengines 302-310 generates an individual impact score based on differentmodels or algorithms related to the virtual machine characteristicsprovided by the data collection engine 202. For example, where theimpact prediction engine 204 includes five different prediction engines302-310, the impact prediction engine 204 may generate five differentimpact scores (e.g., impact sub-scores) ranging from low to high. In oneor more embodiments, the prediction engine 204 combines the differentimpact scores to generate a combined impact score reflective of apredicted impact of live-migrating the virtual machine associated with acorresponding set of virtual machine characteristics.

For example, as shown in FIG. 3, the impact prediction engine 204includes a combiner 312 including a model or algorithm for combiningmultiple discrete impact scores (e.g., two or more impact sub-scores)from the different prediction engines 302-310 within the impactprediction engine 204. The combiner 312 can accumulate the impact scoresin a variety of ways. For example, the combiner 312 can generate aweighted sum of impact scores by applying one or more weights to theindividual impact scores to grant a higher weight to those impactsscores that may have a higher influence on projected impact (e.g.,blackout and brownout impact scores) while applying a lower weight toone or more impact scores that have a lesser influence on the projectedimpact. Different impact scores may have higher or lower weightsdepending on a purpose or triggering circumstance for live-migrating thevirtual machine(s).

Moreover, while FIG. 3 illustrates one example in which each of theimpact scores are provided to a combiner 312 to sum or otherwise combinethe discrete impact scores, the impact prediction engine 204 may have adifferent series of inputs in which impact scores for one or more of theprediction engines depend from one or more impact scores determined byother prediction engines. For example, where the lifetime predictionengine 306 identifies that a virtual machine is set to expire in thenear future (e.g., prior to a time when the virtual machine would bescheduled to live-migrate), an associate impact score may be weightedconsiderably higher than any other discrete impact score. Alternatively,where the lifetime prediction engine 306 determines that a virtualmachine is not set to expire (or not set to expire for a long time), acorresponding impact score may have a minimal influence (e.g., zeroinfluence) on a predicted impact of live-migrating.

Alternatively, in one or more embodiments, impact prediction engine 204only utilizes a subset of the different prediction engines 302-310. Forexample, where a virtual machine lifetime is not a factor (e.g., avirtual machine is not predicted to ever terminate), the impactprediction engine 204 may exclude the lifetime prediction engine 306from determining and providing a corresponding impact sub-score to thecombiner 312 to determine a combined impact score. In addition, theimpact prediction engine 204 may utilize different subsets of thedifferent prediction engines 302-310 based on a condition that triggersthe low-impact live-migration system 104 to determine which virtualmachine(s) to live-migrate (e.g., detecting fewer than a thresholdnumber of empty server nodes, predicting failure of a server node,etc.).

Accordingly, while one or more embodiments of the combiner 312 includean algorithm or model for determining a combined impact score based on aweighted sum, the combiner 312 may consider additional factors and/orimplement a variety of algorithms and models to accurately determine aprojected impact of live-migrating a virtual machine. In addition, thecombiner 312 may consider an application or goal of the live-migrationin accordance with one or more examples described above (e.g., loadbalancing, increasing fault tolerance, de-fragmentation, evacuating aserver node based on predicted failure or a scheduled OS update, etc.)

Upon generating the predicted impact score (e.g., based on a combinationof the impact scores generated by the different prediction engines302-310, the impact prediction engine 204 provides the impact score(s)for one or more virtual machines to the migration engine 206. Inaccordance with one or more examples described above, the migrationengine 206 can selectively identify virtual machines to live-migratebased on corresponding impact scores. In particular, the migrationengine 206 may identify a combination of one or more virtual machines(e.g., a subset of virtual machines from a collection of virtualmachines) to live-migrate based on impact scores.

In addition to generally identifying one or more candidate virtualmachines (or candidate nodes) for live-migration, the migration engine206 can additional generate a migration schedule indicating both thecandidate virtual machines selected for live-migration and a timing oflive-migrating one or more candidate virtual machines. For example, inone or more embodiments, the migration engine 206 generates a migrationschedule that identifies one or more virtual machines associated withlow impact scores and identifies one or more times for a scheduledlive-migration of the virtual machine(s) from one server node to anotherserver node.

The migration engine 206 can schedule the time of the live-migrationbased on one or more prediction scores and/or based on virtual machinecharacteristics indicating usage patterns of the virtual machine(s). Forexample, the migration engine 206 can schedule live-migration of a firstvirtual machine at night where virtual machine characteristics indicatethat the first virtual machine has a lower rate of usage at night thanduring the day. The migration engine 206 can alternatively schedulelive-migration of a second virtual machine during the day where virtualmachine characteristics indicate that the second virtual machine has alower rate of usage during the day than at night. Indeed, the migrationengine 206 can identify multiple virtual machines as well as scheduledifferent times for live-migrating the respective virtual machines.

In addition to identifying virtual machines for live-migration andscheduling a time for initiating the live-migration, the migrationengine 206 can additionally determine a destination server node. Inparticular, the migration engine 206 can identify a destination servernode to receive the virtual machine as part of the live-migration. Thedestination server node may be on the same node cluster as the servernode from which the virtual machine is live-migrating. Alternatively,the destination node may be on a different node cluster within the cloudcomputing system 101.

The migration engine 206 may determine the destination server node basedon a variety of factors. For example, the migration engine 206 mayidentify a destination server node having similar hardware or similar OSas the node from which the virtual machine is migrating. The migrationengine 206 can similarly identify an empty node or a node already havingone or more virtual machines thereon based on a purpose oflive-migrating the virtual machine (e.g., to free up one or more emptynodes or to load balance between different server nodes to allow forscaling of a virtual machine).

FIG. 4 illustrates an example implementation in which a low-impactlive-migration system 104 selectively identifies a subset of virtualmachines from within a node cluster to migrate from corresponding servernodes to destination nodes. For example, as shown in FIG. 4, an examplenode cluster 402 includes server nodes 404 a-404 d. A first server node404 a includes a virtual machine 406 a, a second server node 404 bincludes virtual machines 406 b-c, a third server node 404 c includesvirtual machines 406 d-e, and a fourth server node 404 d includes avirtual machine 406 f.

The low-impact live-migration system 104 may collect and analyze virtualmachine characteristics for each of the virtual machines 406 a-f on thedifferent server nodes 404 a-d of the node cluster 402. The low-impactlive-migration system 104 can further apply an impact prediction engine204 including any number of different types of prediction engines (e.g.,prediction engines 302-310) to determine impact scores for each of thevirtual machines 406 a-f.

While one or more embodiments involve applying the impact predictionengine 204 to determine impact scores for each of the virtual machines,the low-impact live-migration system 104 may selectively apply theimpact prediction engine 204 to a subset of the virtual machines basedon a desired goal or reason for triggering the live-migration. Forexample, where the low-impact live-migration system 104 is initiatingthe analysis and live-migration based on a goal of freeing upallocatable space on the second server node 404 b and the third servernode 404 c and the decision of live-migrating the virtual machine comesdown to determining which virtual machine from those two server nodesshould be migrated to achieve the lowest customer impact, the low-impactlive-migration system 104 may limit analysis of virtual machinecharacteristics to determine impact scores for only those virtualmachines on the server nodes of interest (e.g., virtual machines 406b-e) without applying the impact prediction engine 204 to other virtualmachines on other server nodes within the node cluster 402. As anotherexample, where one or more virtual machines are designated or identifiedas high-priority or otherwise tagged for not being live-migrated (exceptunder potential node-failure conditions), the low-impact live-migrationsystem 104 may similarly forego analysis of those virtual machines insome cases.

As shown in FIG. 4, the low-impact live-migration system 104 mayidentify the second virtual machine 406 b on the second server node 404b and the fourth virtual machine 406 d on the third server node 404 c ascandidate virtual machines for live-migration. The low-impactlive-migration system 104 can additionally identify the first servernode 404 a and the fourth server node 404 d as destination nodes for thelive-migration. The low-impact live-migration system 104 may determinethe destination nodes for each of the respective candidate virtualmachines 406 b, 406 d based on the virtual machine characteristics. Forexample, the low-impact live-migration system 104 may determine thatlive-migrating the second virtual machine 406 b to the first server node404 a and live-migrating the fourth virtual machine 406 d to the fourthserver node 404 d provides the better load balance based on the sizes orother characteristics of the respective virtual machines 406 b, 4046 d.

As shown in FIG. 4, after initiating and performing the live-migration,the node cluster 402 includes the first server node 404 a including thefirst virtual machine 406 a and the second virtual machine 406 b, thesecond server node 404 b including the third virtual machine 406 c, thethird server node 404 c including the fifth virtual machine 406 e, andthe fourth server node 404 d including the fourth virtual machine 406 dand the sixth virtual machine 406 f.

FIG. 5 illustrates an additional example implementation in which thelow-impact live-migration system 104 selectively identifies a subset ofvirtual machines from a node cluster to migrate from correspondingserver nodes to destination server nodes. For example, as shown in FIG.5 an example node cluster 502 includes server nodes 504 a-d. The firstserver node 504 a includes a first virtual machine 506 a, a secondserver node 504 b includes second and third virtual machines 506 b-c, athird server node 504 c is an empty node, and a fourth server node 504 dhas a fourth virtual machine.

Similar to one or more examples described above, the low-impactlive-migration system 104 can analyze virtual machine characteristicsfor some or all of the virtual machines 506 a-d to determine impactscores for the respective virtual machines. In this example, thelow-impact live-migration system 104 may identify a set of virtualmachines for migration based on instructions to maintain a thresholdnumber of empty nodes, a predicted failure of one or more server nodes,a scheduled OS update, or other circumstances that necessitateevacuating one or more of the server nodes 504 a-d.

In the example shown in FIG. 5, the low-impact live-migration system 104may determine impact scores for the virtual machines 506 a-d to identifyone of the server nodes having virtual machines thereon that have alower predicted impact of live-migrating the virtual machines from thatserver node. For example, the low-impact live-migration system 104 maydetermine that a combined impact score (e.g., a sum or other combinationof the impact scores) for the second virtual machine 506 b and the thirdvirtual machine 506 c on the second server node 504 b is less than animpact score for the first virtual machine 506 a or the fourth virtualmachine 506 d. Accordingly, the low-impact live-migration system 104 candetermine that live-migrating the second virtual machine 506 b and thethird virtual machine 506 c will have less of an impact than migratingeither of the first virtual machine 506 a or the fourth virtual machine506 d individually.

As mentioned above, in addition to generally selectively identifying asubset of virtual machine to live-migrate, the low-impact live-migrationsystem 104 can additionally generate a migration schedule or otherwiseidentify a time for live-migrating the selected virtual machines tofurther minimize an impact of the live-migration. For example, thelow-impact live-migration system 104 can identify a first time forlive-migrating the second virtual machine 506 b and a second time (e.g.,a different day or time of day) for live-migrating the third virtualmachine 506 c.

As shown in FIG. 5, after initiating and performing the live-migration,the node cluster 502 includes the first server node 504 a including thefirst virtual machine 506 a and the second virtual machine 506 b and thefourth server node 504 d including the third virtual machine 506 c andthe fourth virtual machine 506 d. In addition, after performing thelive-migration, the node cluster 502 includes empty nodes 504 b-c, whichmay correspond to a predetermined number (e.g., a minimum or desiredthreshold) of empty nodes on the node cluster 502.

As noted above, the low-impact live-migration system 104 can evacuateone or more nodes under various circumstances. For example, thelow-impact live-migration system 104 can identify a server node from anode cluster to evacuate in order to have a minimum threshold or otherdesired number of empty nodes. As another example, the low-impactlive-migration system 104 can identify an empty node for evacuationbased on a scheduled OS upgrade that necessitates evacuating the servernode within a limited window of time. In one or more embodiments, thelow-impact live-migration system 104 receives an indication or predictsan instance of node failure and evaluates the virtual machines todetermine a time within the limited window when live-migrating thevirtual machines will have a minimal impact on customers.

In any of the above and other circumstances, the low-impactlive-migration system 104 may receive an indication or determine a levelof urgency associated with live-migrating one or more virtual machines.For example, where a node cluster of thirty server nodes optimallyincludes five empty nodes, the low-impact live-migration system 104 maydetermine that a desired threshold of empty nodes if five. Accordingly,where the node cluster only has four empty nodes, the low-impactlive-migration system 104 may begin the process of evaluating virtualmachine characteristics to selectively identify a server node toevacuate based on impact scores of virtual machines thereon.

In another example, where the node cluster has only one (or zero) emptynodes (or multiple nodes less than the desired number of empty nodes),the low-impact live-migration system 104 may similarly evaluate theserver nodes to identify a candidate node for migration with anadditional measure of urgency. For example, the low-impactlive-migration system 104 may determine not to live-migrate one or morevirtual machines even where the node cluster does not have a desirednumber of empty nodes based on a determination that live-migrating a setof virtual machines to empty a server node will have a significantlyunfavorable impact. However, over time, as additional empty nodes havevirtual machines implemented thereon, the low-impact live-migrationsystem 104 may determine that emptying one or more server nodes hasbecome more urgent and decide to live-migrate virtual machines having ahigher predicted impact based on a higher urgency to empty the servernodes.

In other words, the low-impact live-migration system 104 may determinewhether to live-migrate one or more virtual machines based on impactscore(s) in addition to a threshold impact in accordance withcircumstances that trigger live-migration of the virtual machine. Inaddition, based on changing circumstances that may change a level ofurgency, the low-impact live-migration system 104 may determine that adifferent threshold level of impact should apply in deciding whether tolive-migrate one or more candidate virtual machines. An exampleimplementation is described in further detail in connection with FIG. 6.

FIG. 6 illustrates an example series of acts 600 for selectivelyidentifying one or more virtual machines for live-migration based ondetermined impact scores and further in view of a threshold impact. Theseries of acts 600 shown in FIG. 6 may be performed by the low-impactlive-migration system 104 in accordance with one or more embodimentsdescribed herein. For example, as shown in FIG. 6, the low-impactlive-migration system 104 can perform an act 610 of collecting virtualmachine data. The act 610 of collecting the virtual machine data mayinclude similar features as discussed above with identifying virtualmachine characteristics by the data collection engine 202 discussedabove.

The low-impact live-migration system 104 can further perform an act 620of determining an impact threshold. As mentioned above, the low-impactlive-migration system 104 may determine an impact threshold based on acurrent resource allocation state of a node cluster or a server node,which may involve an analysis of characteristics of a server node, nodecluster, or conditions that trigger the decision to live-migrate thevirtual machine(s). For example, the low-impact live-migration system104 may determine an impact threshold based on a number of empty nodesin a node cluster compared to a desired number or minimum number ofserver nodes for the node cluster. Accordingly, the low-impactlive-migration system 104 may identify a high impact threshold where thenode cluster already has a desired number or minimum threshold of emptynodes thereon. Alternatively, the low-impact live-migration system 104may lower the impact threshold as the number of empty nodes decreases orcomes closer to a minimum threshold of virtual machines.

As another example, the low-impact live-migration system 104 maydetermine an impact threshold based on a time period within whichlive-migration of the virtual machine needs to occur. For example, wherethe low-impact live-migration system 104 determines or receives anindication of an estimated failure of a server node, the low-impactlive-migration system 104 may determine the impact threshold based on anamount of time between the present and the estimated failure of theserver node. Accordingly, where the estimated failure of the server nodeis multiple days away, the low-impact live-migration system 104 may havea higher impact threshold (e.g., indicating a lower sense of urgency)than if the estimated failure of the server node were only a few hoursaway. Therefore, the low-impact live-migration system 104 can determinea different impact threshold as well as modify the impact threshold overtime based on characteristics of virtual machines, server nodes, a nodecluster, and/or circumstances associated with triggering thelive-migration of the virtual machine(s).

As another example, the low-impact live-migration system 104 maydetermine an impact threshold based on a current fault tolerance of anode cluster or select server nodes. For example, where a node clusterincludes a first server node that is overloaded with virtual machines,the low-impact live-migration system 104 may apply a different migrationthreshold when determining whether to migrate virtual machines from thefirst server node than a second server node that is less overloaded andhas a higher fault tolerance than the first server node. Accordingly, inone or more embodiments, the low-impact live-migration system 104 candetermine a different impact threshold based on a detected faulttolerance for a given server node or specific virtual machine(s).

As a further example, the low-impact live-migration system 104 candetermine an impact threshold based on an owner of the virtual machineand/or instructions received from the owner. For instance, thelow-impact live-migration system 104 can assign a high or low impactthreshold to different owners or customers. Further, in one or moreimplementations, the low-impact live-migration system 104 can assign orotherwise determine a threshold in response to receiving instructions(e.g., a selected setting or marker) indicating or marking that virtualmachine as risk-averse, and that an appropriately high or low impactthreshold should be associated with the virtual machine.

As further shown in FIG. 6, the low-impact live-migration system 104 canperform an act 630 of determining predicted impact scores. Inparticular, the low-impact live-migration system 104 can determinepredicted impact scores for any number of virtual machines on a servernode and/or across multiple server nodes of the cloud computing system101. In one or more embodiments, the low-impact live-migration system104 determines the predicted impact score(s) by applying an impactprediction engine 204 to virtual machine characteristics in accordancewith one or more examples described above.

The low-impact live-migration system 104 can additionally perform an act640 of identifying a candidate virtual machine for live-migration. Thismay involve identifying a virtual machine from a set of virtual machineshaving the lowest impact score. For example, the low-impactlive-migration system 104 may identify a candidate virtual machine froma set of virtual machines of a server node to live-migrate based on adetermination that the candidate virtual machine has a lower impactscore than other virtual machines on the server node. The low-impactlive-migration system 104 can similarly identify a candidate virtualmachine from a set of virtual machines across multiple server nodes.

While the low-impact live-migration system 104 may identify a candidatevirtual machines, in one or more embodiments, the low-impactlive-migration system 104 identifies a candidate server node based onimpact scores for one or multiple virtual machines thereon. For example,the low-impact live-migration system 104 may identify a candidate servernode from a plurality of server nodes based on a determination that acombination of impact scores (e.g., a sum of impact scores) for virtualmachines on the candidate server node is less than a combined impactscore of one or more virtual machines on other server nodes from theplurality of server nodes.

As shown in FIG. 6, the low-impact live-migration system 104 canadditionally perform an act 650 of determining whether the predictedimpact (e.g., the impact score) is greater than an impact threshold(e.g., a threshold impact score). If the predicted impact is not greaterthan the impact threshold, the low-impact live-migration system 104 mayforego live-migrating the candidate virtual machine (or candidate servernode) and instead continue collecting virtual machine data to determineif one or more of the impact threshold or impact score(s) changes overtime.

Alternatively, where the low-impact live-migration system 104 determinesthat the predicted impact is greater than the impact threshold, thelow-impact live-migration system 104 can perform an act 660 ofinitiating live-migration of the candidate virtual machine. In one ormore embodiments, the low-impact live-migration system 104 utilizes themigration engine 206 to identify a time to initiate or otherwise carryout the live-migration. The low-impact live-migration system 104 canadditionally determine a destination node on the same or different nodecluster to receive the candidate virtual machine.

Turning now to FIGS. 7-8, these figures illustrate example flowchartsincludes series of acts for selectively live-migrating virtual machinesbetween server devices of a cloud computing system 101. While FIGS. 7-8illustrate acts according to one or more embodiments, alternativeembodiments may omit, add to, reorder, and/or modify any of the actsshown in FIGS. 7-8. The acts of FIGS. 7-8 can be performed as part of amethod. Alternatively, a non-transitory computer readable medium cancomprise instructions that, when executed by one or more processors,cause a server device to perform the acts of FIGS. 7-8. In still furtherembodiments, a system can perform the acts of FIGS. 7-8.

For example, the series of acts 700 includes an act 720 of identifying aplurality of computing containers or virtual machines on a cloudcomputing system. For example, the act 720 can include identifying aplurality of virtual machines on a plurality of server devices.

The series of acts 700 can further include an act 740 of evaluatingvirtual machine characteristics to determine a plurality of impactscores for the plurality of virtual machines. For example, the act 740can include evaluating virtual machine characteristics of the pluralityof virtual machines to determine a plurality of impact scores for theplurality of virtual machines where the plurality of impact scoresincludes indications of predicted impact of live-migrating the pluralityof virtual machines.

In one or more embodiments, evaluating the virtual machinecharacteristics of the plurality of virtual machines to determine aplurality of impact scores includes applying a prediction engine to theplurality of virtual machines where the prediction engine is trained todetermine impact scores for virtual machines based on associated virtualmachine characteristics.

For example, applying the prediction engine to the plurality of virtualmachines may include applying a blackout prediction engine trained topredict an estimated blackout time for migrating a given virtual machinebased on a set of virtual machine characteristics for the given virtualmachine where the estimated blackout time includes an estimated durationof time that the virtual machine will not execute any codes orinstructions or that the client device will be disconnected fromaccessing the given virtual machine. Applying the prediction engine todetermine impact scores can further include determining the plurality ofimpact scores based on predicted blackout times for the plurality ofvirtual machines.

As another example, applying the prediction engine to the plurality ofvirtual machines may include applying a brownout prediction enginetrained to predict an estimated brownout time for migrating a given datacontained based on a set of virtual machine characteristics for thegiven virtual machine where the estimated brownout time includes anestimated duration of time that the given data container will providelimited performance. Applying the prediction engine to determine impactscores can further include determining the plurality of impact scoresbased on predicted brownout times for the plurality of virtual machines.

As another example, applying the prediction engine to the plurality ofvirtual machines may include determining impact sensitivity for theplurality of virtual machines where impact sensitivity for a givenvirtual machine includes an indication of sensitivity of the givenvirtual machine is to an interruption of a connection between a clientdevice and the given virtual machine.

In one or more embodiments, the prediction engine can be trained topredict an impact score associated with live-migrating container bydetermining two or more impact sub-scores for the given virtual machine.The two or more impact sub-scores may include two or more of: a firstpredicted impact sub-score associated with a predicted blackout timewhere the blackout time includes an estimated duration of time that thegiven virtual machine will not execute any codes or instructions or thatthe client device will be disconnected from accessing the given virtualmachine, a second predicted impact sub-score associated with a predictedbrownout time where the predicted brownout time includes an estimatedduration of time that the client device will have limited access to thegiven virtual machine, a third predicted impact sub-score associatedwith an impact sensitivity where the impact sensitivity includes anindication of how sensitive the given virtual machine is to aninterruption of a connection between a client device and the givenvirtual machine, and a fourth predicted impact sub-score associated witha lifetime of the virtual machine where the lifetime of the virtualmachine includes an indication of when the virtual machine is expectedto expire. The prediction engine may further be trained to combine thetwo or more predicted impact sub-scores to determine the impact scoreassociated with live-migrating the given virtual machine.

The series of acts 700 can further include an act 760 of identifying acandidate virtual machine for live-migration based on an impact scorefor the candidate virtual machine. For example, the act 760 can includeidentifying a candidate virtual machine for live-migration from theplurality of virtual machines based on an impact score from theplurality of impact scores corresponding to the virtual machine.Identifying the candidate virtual machine for live-migration may includeidentifying the candidate virtual machine based on a determination thatthe impact score corresponding to the virtual machine is less than anadditional impact score corresponding to an additional virtual machinefrom the plurality of virtual machines.

The series of acts 700 further includes an act 780 of initiatinglive-migration of the candidate virtual machine. For example, the act780 can include initiating live-migration of the candidate virtualmachine to a destination server device. In one or more embodiments, theseries of acts 700 further includes identifying a threshold impactscore. The series of acts 700 can further include initiatinglive-migration of the candidate virtual machine to the destinationserver device based on determining that the impact score for thecandidate node is less than the threshold impact score.

The series of acts 800 shown in FIG. 8 includes an act 820 ofidentifying a plurality of server devices including a plurality ofvirtual machines. For example, the act 820 can include identifying aplurality of server devices including a plurality of virtual machines.

The series of acts 800 can further include an act 840 of evaluatingvirtual machine characteristics to determine a plurality of impactscores associated with live-migrating the plurality of virtual machines.For example, the act 840 may include evaluating virtual machinecharacteristics of the plurality of virtual machines to determine aplurality of impact scores associated with live-migrating the pluralityof virtual machines where an impact score of a virtual machine comprisesan indication of a predicted impact of live-migrating the virtualmachine.

In one or more embodiments, applying the prediction engine to thevirtual machines includes applying a blackout prediction model trainedto predict a range of blackout times from a plurality of ranges ofblackout times based on a set of virtual machine characteristics for agiven virtual machine where the plurality of ranges of blackout timesincluding ranges of durations of time that the given virtual machinewill not execute any codes or instructions or that the client device ispredicted to be disconnected from the given virtual machine. Theplurality of ranges of blackout times may include ranges of durations ona log linear scale.

In one or more embodiments, applying the prediction engine to thevirtual machines includes applying a blackout prediction model trainedto predict a range of blackout times from a plurality of ranges ofblackout times based on a set of virtual machine characteristics for agiven virtual machine where the plurality of ranges of blackout timesinclude ranges of durations of time that a client device is predicted tobe disconnected from the given virtual machine. The plurality of rangesof brownout times may include ranges of durations on a log linear scale.

The series of acts 800 can further include an act 860 of identifying acandidate server device from the plurality of server devices based theplurality of impact scores. For example, the act 860 may includeidentifying a candidate server device from the plurality of serverdevices based on one or more impact scores for one or more virtualmachines from the plurality of virtual machines on the candidate server.In one or more embodiments, identifying the candidate server deviceincludes determining a combined impact score for the candidate serverdevice based on a combination of the one or more impact scores for theone or more virtual machines on the candidate server. Identifying thecandidate server may further include determining that the combinedimpact score for the candidate server device is less than one or morecombined impact scores for one or more additional server devices fromthe plurality of server devices.

The series of acts 800 can further include an act 880 of initiatinglive-migration of the one or more virtual machines from the candidateserver device. For example, the act 880 may include initiatinglive-migration of the one or more virtual machines from the candidateserver device to a destination server device. The series of acts 800 canfurther include determining that the one or more impact scores for theone or more virtual machines from the plurality of virtual machines isless than a threshold impact score. The series of acts 800 canadditionally include initiating live-migration of the one or morevirtual machines based determining that the one or more impact scoresfor the one or more virtual machines from the plurality of virtualmachines is less than a threshold impact score. In one or moreembodiments, the threshold impact score is based on a number of emptyserver devices from the plurality of server devices.

While one or more embodiments described herein relate specifically tosystems, methods, and computer-readable media for live-migrating virtualmachines or computing containers between server nodes, similar featuresand functionality can relate to more generally live-migrating virtualservices between nodes of the cloud computing system. For example, theacts discussed in connection with FIGS. 7 and 8 may involve identifyingvirtual services including virtual machines and/or computing containers,evaluating characteristics of the virtual services to determine impactscores (e.g., including applying a prediction engine to the virtualservices), identifying candidate virtual service(s) for live-migrationbased on the impact scores, and initiating live-migration of thecandidate virtual service(s) in accordance with one or more embodimentsdescribed herein.

FIG. 9 illustrates certain components that may be included within acomputer system 900. One or more computer systems 900 may be used toimplement the various devices, components, and systems described herein.

The computer system 900 includes a processor 901. The processor 901 maybe a general purpose single- or multi-chip microprocessor (e.g., anAdvanced RISC (Reduced Instruction Set Computer) Machine (ARM)), aspecial purpose microprocessor (e.g., a digital signal processor (DSP)),a microcontroller, a programmable gate array, etc. The processor 901 maybe referred to as a central processing unit (CPU). Although just asingle processor 901 is shown in the computer system 900 of FIG. 9, inan alternative configuration, a combination of processors (e.g., an ARMand DSP) could be used.

The computer system 900 also includes memory 903 in electroniccommunication with the processor 901. The memory 903 may be anyelectronic component capable of storing electronic information. Forexample, the memory 903 may be embodied as random access memory (RAM),read-only memory (ROM), magnetic disk storage media, optical storagemedia, flash memory devices in RAM, on-board memory included with theprocessor, erasable programmable read-only memory (EPROM), electricallyerasable programmable read-only memory (EEPROM) memory, registers, andso forth, including combinations thereof.

Instructions 905 and data 907 may be stored in the memory 903. Theinstructions 905 may be executable by the processor 901 to implementsome or all of the functionality disclosed herein. Executing theinstructions 905 may involve the use of the data 907 that is stored inthe memory 903. Any of the various examples of modules and componentsdescribed herein may be implemented, partially or wholly, asinstructions 905 stored in memory 903 and executed by the processor 901.Any of the various examples of data described herein may be among thedata 907 that is stored in memory 903 and used during execution of theinstructions 905 by the processor 901.

A computer system 900 may also include one or more communicationinterfaces 909 for communicating with other electronic devices. Thecommunication interface(s) 909 may be based on wired communicationtechnology, wireless communication technology, or both. Some examples ofcommunication interfaces 909 include a Universal Serial Bus (USB), anEthernet adapter, a wireless adapter that operates in accordance with anInstitute of Electrical and Electronics Engineers (IEEE) 802.11 wirelesscommunication protocol, a Bluetooth wireless communication adapter, andan infrared (IR) communication port.

A computer system 900 may also include one or more input devices 911 andone or more output devices 913. Some examples of input devices 911include a keyboard, mouse, microphone, remote control device, button,joystick, trackball, touchpad, and lightpen. Some examples of outputdevices 913 include a speaker and a printer. One specific type of outputdevice that is typically included in a computer system 900 is a displaydevice 915. Display devices 915 used with embodiments disclosed hereinmay utilize any suitable image projection technology, such as liquidcrystal display (LCD), light-emitting diode (LED), gas plasma,electroluminescence, or the like. A display controller 917 may also beprovided, for converting data 907 stored in the memory 903 into text,graphics, and/or moving images (as appropriate) shown on the displaydevice 915.

The various components of the computer system 900 may be coupledtogether by one or more buses, which may include a power bus, a controlsignal bus, a status signal bus, a data bus, etc. For the sake ofclarity, the various buses are illustrated in FIG. 9 as a bus system919.

The techniques described herein may be implemented in hardware,software, firmware, or any combination thereof, unless specificallydescribed as being implemented in a specific manner. Any featuresdescribed as modules, components, or the like may also be implementedtogether in an integrated logic device or separately as discrete butinteroperable logic devices. If implemented in software, the techniquesmay be realized at least in part by a non-transitory processor-readablestorage medium comprising instructions that, when executed by at leastone processor, perform one or more of the methods described herein. Theinstructions may be organized into routines, programs, objects,components, data structures, etc., which may perform particular tasksand/or implement particular data types, and which may be combined ordistributed as desired in various embodiments.

The steps and/or actions of the methods described herein may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isrequired for proper operation of the method that is being described, theorder and/or use of specific steps and/or actions may be modifiedwithout departing from the scope of the claims.

The term “determining” encompasses a wide variety of actions and,therefore, “determining” can include calculating, computing, processing,deriving, investigating, looking up (e.g., looking up in a table, adatabase or another data structure), ascertaining and the like. Also,“determining” can include receiving (e.g., receiving information),accessing (e.g., accessing data in a memory) and the like. Also,“determining” can include resolving, selecting, choosing, establishingand the like.

The terms “comprising,” “including,” and “having” are intended to beinclusive and mean that there may be additional elements other than thelisted elements. Additionally, it should be understood that referencesto “one embodiment” or “an embodiment” of the present disclosure are notintended to be interpreted as excluding the existence of additionalembodiments that also incorporate the recited features. For example, anyelement or feature described in relation to an embodiment herein may becombinable with any element or feature of any other embodiment describedherein, where compatible.

The present disclosure may be embodied in other specific forms withoutdeparting from its spirit or characteristics. The described embodimentsare to be considered as illustrative and not restrictive. The scope ofthe disclosure is, therefore, indicated by the appended claims ratherthan by the foregoing description. Changes that come within the meaningand range of equivalency of the claims are to be embraced within theirscope.

What is claimed is:
 1. A method for live-migrating virtual servicesbetween server nodes, comprising: determining impact scores for aplurality of virtual machines on a cloud computing system, whereindetermining the impact scores includes, for each virtual machine fromthe plurality of virtual machines, determining an impact scoreindicating a predicted impact of live-migrating a virtual machine basedon one or more behavior characteristics of the virtual machine;identifying candidate virtual machines for live-migration based on apolicy of the cloud computing system associated with increasingavailability of computing resources on server devices of the cloudcomputing system, wherein identifying the candidate virtual machinesincludes identifying a subset of virtual machines from the plurality ofvirtual machines based on associated impact scores for the subset ofvirtual machines being less than impact scores for other virtualmachines from the plurality of virtual machines; and initiatinglive-migration of the candidate virtual machines to one or moredestination server devices.
 2. The method of claim 1, wherein the one ormore behavior characteristics includes characteristics of an applicationrunning on the virtual machine.
 3. The method of claim 1, whereindetermining the impact score for each virtual machine from the pluralityof virtual machines includes: applying a brownout prediction engine tothe plurality of virtual machines, wherein the brownout predictionengine is trained to predict an estimated brownout time forlive-migrating each virtual machine from the plurality of virtualmachines, wherein the estimated brownout time includes an estimatedduration of time that a virtual machine will provide limited performancewhile still maintaining a level of client access to the virtual machine;and wherein determining the impact score for the virtual machine isbased on a sensitivity of the virtual machine to the estimated brownouttime.
 4. The method of claim 1, wherein determining the impact score foreach virtual machine from the plurality of virtual machines includes:identifying lifetimes of the plurality of virtual machines, thelifetimes including indications of when the plurality of virtualmachines are expected to expire; and wherein determining the impactscore for the virtual machine is based on an associated lifetime of thevirtual machine.
 5. The method of claim 4, wherein identifying lifetimesof the plurality of virtual machines includes identifying times at whichone or more of the virtual machines are scheduled to expire.
 6. Themethod of claim 4, wherein determining the impact score for the virtualmachine includes: comparing the associated lifetime to a time when thevirtual machine would be scheduled for live-migration; and determiningthe impact score for the virtual machine based on comparing theassociated lifetime to the time when the virtual machine would bescheduled for live-migration.
 7. The method of claim 6, whereindetermining the impact score includes: if the associated lifetime isearlier than the time when the virtual machine would be scheduled forlive-migration, associating the virtual machine with a first impactscore; and if the associated lifetime is greater than the time when thevirtual machine would be schedule for live-migration, associating thevirtual machine with a second impact score.
 8. The method of claim 7,wherein the first impact score is greater than the second impact scorebased on the associated lifetime being earlier than the time when thevirtual machine would be scheduled for live-migration.
 9. The method ofclaim 1, wherein the one or more behavior characteristics includes oneor more of a daily or weekly utilization pattern for the virtualmachine.
 10. The method of claim 9, wherein determining the impact scorefor each virtual machine includes: identifying a time at which a virtualmachine would be scheduled for live-migration; and determining, based onthe daily or weekly utilization pattern, the impact score based on apredicted utilization for the virtual machine at the time when thevirtual machine would be scheduled for live-migration.
 11. The method ofclaim 1, wherein the one or more behavior characteristics includes twoor more of: characteristics of an application running on the virtualmachine; a sensitivity of the virtual machine to an estimated brownouttime, the estimated brownout time including an estimated duration oftime that a virtual machine will provide limited performance while stillmaintaining a level of access to the virtual machine; a lifetime of thevirtual machine, the lifetime including an indication of when thevirtual machine is expected to expire; and a daily or weekly utilizationpattern for the virtual machine.
 12. The method of claim 1, furthercomprising: identifying a threshold impact score; and whereinidentifying the candidate virtual machines for live-migration is basedon determining that the impact scores for the subset of virtual machinesare greater than the threshold impact score.
 13. A system forlive-migrating virtual services between server nodes, comprising: one ormore processors; memory in electronic communication with the one or moreprocessors; and instructions stored in the memory, the instructionsbeing executable by the one or more processors to: determine impactscores for a plurality of virtual machines on a cloud computing system,wherein determining the impact scores includes, for each virtual machinefrom the plurality of virtual machines, determining an impact scoreindicating a predicted impact of live-migrating a virtual machine basedon one or more behavior characteristics of the virtual machine; identifycandidate virtual machines for live-migration based on a policy of thecloud computing system associated with increasing availability ofcomputing resources on server devices of the cloud computing system,wherein identifying the candidate virtual machines includes identifyinga subset of virtual machines from the plurality of virtual machinesbased on associated impact scores for the subset of virtual machinesbeing less than impact scores for other virtual machines from theplurality of virtual machines; and initiate live-migration of thecandidate virtual machines to one or more destination server devices.14. The system of claim 13, wherein the one or more behaviorcharacteristics includes one or more of: characteristics of anapplication running on the virtual machine; a sensitivity of the virtualmachine to an estimated brownout time, the estimated brownout timeincluding an estimated duration of time that a virtual machine willprovide limited performance while still maintaining a level of access tothe virtual machine; a lifetime of the virtual machine, the lifetimeincluding an indication of when the virtual machine is expected toexpire; or a daily or weekly utilization pattern for the virtualmachine.
 15. The system of claim 13, wherein determining the impactscore for each virtual machine from the plurality of virtual machinesincludes: applying a brownout prediction engine to the plurality ofvirtual machines, wherein the brownout prediction engine is trained topredict an estimated brownout time for live-migrating each virtualmachine from the plurality of virtual machines, wherein the estimatedbrownout time includes an estimated duration of time that a virtualmachine will provide limited performance while still maintaining a levelof client access to the virtual machine; and wherein determining theimpact score for the virtual machine is based on a sensitivity of thevirtual machine to the estimated brownout time.
 16. The system of claim13, wherein determining the impact score for each virtual machine fromthe plurality of virtual machines includes: identifying lifetimes of theplurality of virtual machines, the lifetimes including indications ofwhen the plurality of virtual machines are expected to expire; andwherein determining the impact score for the virtual machine is based onan associated lifetime of the virtual machine.
 17. The system of claim16, wherein determining the impact score for the virtual machineincludes: comparing the associated lifetime to a time when the virtualmachine would be scheduled for live-migration; and determining theimpact score for the virtual machine based on comparing the associatedlifetime to the time when the virtual machine would be scheduled forlive-migration.
 18. The system of claim 17, wherein determining theimpact score includes: if the associated lifetime is earlier than thetime when the virtual machine would be scheduled for live-migration,associating the virtual machine with a first impact score; and if theassociated lifetime is greater than the time when the virtual machinewould be schedule for live-migration, associating the virtual machinewith a second impact score, wherein the first impact score is greaterthan the second impact score.
 19. The system of claim 13, wherein theone or more behavior characteristics includes one or more of a daily orweekly utilization pattern for the virtual machine.
 20. The system ofclaim 19, wherein determining the impact score for each virtual machineincludes: identifying a time at which a virtual machine would bescheduled for live-migration; and determining, based on the daily orweekly utilization pattern, the impact score based on a predictedutilization for the virtual machine at the time when the virtual machinewould be scheduled for live-migration.